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Abstract. Any traditional engineering field has metrics to rigorously 
assess the quality of their products. Engineers know that the output 
must satisfy the requirements, must comply with the production and 
market rules, and must be competitive. 

Professionals in the new field of software engineering started a few years 
ago to define metrics to appraise their product: individual programs and 
software systems. This concern motivates the need to assess not only the 
outcome but also the process and tools employed in its development. In 
this context, assessing the quality of programming languages is a legit- 
imate objective; in a similar way, it makes sense to be concerned with 
models and modeling approaches, as more and more people start the 
software development process by a modeling phase. 

In this paper we introduce and motivate the assessment of models quality 
in the Software Development cycle. After the general discussion of this 
topic, we focus the attention on the most popular modeling language - 
the UML - presenting metrics. Through a Case-Study, we present and 
explore two tools. To conclude we identify what is still lacking in the 
tools side. 

Keywords: Modeling Languages, Software/Language Quality, Software/Lan- 
guage Metrics, UML 

1 Introduction 

Models are a representation of reality aiming at the simplification of some com- 
plex objects: they are built so that we can better understand the system being 
developed. They allow us to specify the structure and behavior of a system, 
providing the guidance lines/blueprints for constructing a system, and finally, 
they document the decisions taken for a given system. Specifying means building 
models that are precise, unambiguous and complete. 

Some models arc best described tcxtually, other graphically All interesting 
systems exhibit structures that transcend what can be represented in a program- 
ming language. 



A modeling language is a language whose vocabulary and rules focus on the 
conceptual and physical representation of a system. One the one hand, one can 
produce a strict formal specification of the system, which allows us to reason 
over the system proprieties, without running the system. On the other hand, 
one can follow a pragmatic approach, using a diagrammatic specification of the 
system, not allowing us to reason over programs, but deriving programs from the 
model specification. That aside, when assessing a modeling language we might 
need to infer on its quality. 

The main goal of using model/software metrics is to be able to generate quan- 



tifiable measurements from the specifications/software. According to Mil98 
metrics can be used to improve software productivity and quality. The use of 
model metrics is even more important to numerous valuable applications in 
earlier stages of the development process like scheduling, cost estimation, qual- 
ity assurance, and personnel task assignments. Nowadays, this metrics become 
increasingly essential for Software Engineering: they are crucial even for reengi- 
neering processes. In Forward Engineering they are used to measure the software 



quality and estimate cost and effort of software projects FP98 . In the field of 
Software Evolution, they can be both used to identify stable or unstable parts of 
a system as to determine where refactoring can be or have been applied [DDN00| . 
They even can be used for assessing the quality and complexity of software sys- 
tems in Software Reengineering or Reverse Engineering |CC90| . 

When focusing on the field of Object-Oriented (00) systems, many metrics 
have been proposed for assessing the design of a software system. However, most 
of the existing approaches involve the analysis of the source code and cannot be 
applied in earlier stages of the development process. In fact, it is not always sim- 
ple to apply the existing metrics in this earlier stages. As the Unified Modeling 



Language, proposed by Booch, Jacobson and Rumbaugh BRJ05 has became 
a standard for expressing, design and specify 00 systems, applying metrics to 
these models enables an early estimate of development efforts, implementation 
time, complexity and cost of the system under development. 

In this paper, we introduce and discuss the major existing metrics for UML 
models and present a set of tools designed for measuring UML projects. In Sec- 
tion [2] we describe the principal measurements applicable to the most popular 
UML diagrams. Then, in Section [3] we present two of the best tools designed 
to extract metrics from UML models and the results of applying them to a real 
case-study. Then, Section [4] is devoted to the metrics assessment problem. We 
conclude in Section [5] with a final balance of these topics. 



2 Applying Metrics To UML Models 

An UML model can be made from different diagrams, each one with a distinct 
view of the system. We have, in one hand, Use Cases diagrams which expose the 
system functional requirements and how each user interacts with them. They 
are a good overview of what features the system offers to the end user. In the 



other hand, we use Class Diagrams for represent the blueprint of the application 
under the developer perspective: they illustrate which programming components 
a system has and how they related to each other. Package Diagrams describe how 
to group the classes and how these groups relate to each other (package import, 
package merge). Here we present some metrics related to this three fundamental 
diagrams and conclude the section by introducing some metrics for other not 
less important UML diagrams. 



2.1 Object-Oriented Software: CK Metrics 

One of the most popular suites of 00 metrics was proposed by Chidamber and 



Kemerer CK94 to capture different aspects of 00 designs, including complexity, 



coupling and cohesion. As we can see in MP06 , they were posteriorly adapted 
for modeling languages and can be easly applied to UML class diagrams. 

This suite is composed by six metrics: Weighted Methods Per Class (WMC), 
Depth of Inheritance Tree (DIT), Number of Children (NOC), Response For 
a Class (RFC), Coupling between Object Classes (CBO) and, finally, Lack of 
Cohesion in Methods (LCOM). We detail bellow each metric and its features. 

Weighted methods per class (WMC) - This metric regards to the complexity of 
a class method, being equal to the sum of those methods complexities. There 
are two kinds of WMC metrics: 

— WMCli is computed from a class diagram by counting the number of meth- 
ods in that class - considering each method as an unity; 

— WMC CC is computed by counting the number of methods in each class, based 
on the result of the McCabe Cyclomatic Complexity of each method. 

Depth of inheritance tree (DIT) - This metric is equal to the maximum length 
from the class to the root of the inheritance, which could be defined as the depth 
of the class. It is computed by taking the union of all the class diagrams in a 
UML model and traversing the inheritance hierarchy of the class. 

Number of children (NOC) - This metric represents the number of childs and 
descendants of a certain class. Can be obtained gathering all diagrams class, in 
a UML modulation, and checking all the inheritance relations of the class. 

Response for a class (RFC) - This metric measures the number of methods that 
can be invoked by an object of a given class. It can be obtained from a class 
diagram and from behavior diagrams (e.g. sequence diagrams), which can inform 
of several methods of other classes that are invoked by each of the class methods. 

Coupling between object classes (CBO) - Two classes are related if a method 
of a class uses a instance variable or method of another class. Thus, we can 
compute this metric by counting the number of classes to which the class is 
related and counting all kind of references of the attributes and parameters of 



the class methods. Though, it is possible to calculate a more precise value if 
behavioral diagrams are taken into account, since the usage of instance variable 
and invocation methods are additional information. 



Lack of cohesion in methods (LCOM) - It measures the number of sets of in- 
stance variables accessed by every pair of methods of a given class, that has a 
non-empty intersection. For this, is essential to use the information of the usage 
instance variables by the methods of a class - i.e., since a class diagram does not 
have information about the usage, it is required a sequence diagram. 



This set of metrics can be found and cited in several papers - like MP06 
and represent the basis of all the existing metrics for 00 systems. 



2.2 Class Diagram and Package Metrics 

These diagrams are used to describe the types of objects in a system and the 
relationships among them. They describe the structure of a system by showing 
the system classes, their attributes and methods or operations. Their quality 
have a huge impact on the final quality of the software under development, as 
they describe the general model of the system information. 



Marchesi Metrics 



Metric 


Description 


NC 


Number of Classes 


CL1 


Weighted Number of Class 
responsibilities 


CL2 


Weighted Number of Class 
Dependencies 


CL3 


Depth of inheritance tree 


CL4 


Number of immediate sub- 
classes of a given class 


CL5 


Number of distinct class 



Table 1. Marchesi Class Diagram Metrics 



Marchesi Metrics 



Metric 


Description 


NP 


Number of Packages. 


PK1 


Number of Classes 


PK2 


Weighted Number of respon- 




sibilities of a Class 


PK3 


Overall Coupling among 




Packages 



Table 2. Marchesi Package Metrics 



Measures like the Number of Attributes in the Class, the Number of Opera- 
tions in the Class, Number of Inherited Attributes, Number of descents/ancestors 
of a Class, or even the Number of Interfaces Implemented are used both for in- 
dicate the system complexity as for an index of quality. Many works present 

The OOA metrics 



Eic06 , YWG04 



several metrics for this diagrams GPCOO 
defined in Mar98 also contemplate Class Diagrams, as we can see in Table [l] 

In UML, classes can be grouped into Packages to define subsystems or even 
for implementation purposes. The measurement of a package complexity is useful 
to forecast the development efforts of it. For that, we can measure properties 
like the Number of Classes of a Package, the Total Number of Packages in the 



system, or the Number of Interfaces in the Package. Marchesi Mar98 suggests 
several Package Metrics as described in Table [2] for estimate this complexity. 



2.3 Use Case Metrics 



Use Cases Diagrams are graphical representations of entities which interact with 
the system (actors) and operations that the system must perform for them. They 
define a sequence of actions which illustrate a specific way of using the system. 

These diagrams are functional specifications, collected at the beginning of a 
system development process. They are crucial to an early estimate of the system 
complexity and its development efforts, as we can see by the UC metrics defined 
in several works like 



KB02 , MAC05 , RibOl 



In fact, measuring the number of Use Cases, actors and communications 
among them is a good indicator of the system complexity, as well as to quantify 
the relationship between diagrams (i.e. estimates the number of UC that extend 
or include others). 



One remarkable work on this area was performed by Michele Marchesi Mar98 
Table |3] illustrates the Use Case metrics defined on this work. 



Marchesi Metrics 



Metric 


Description 


NA 


Number of actors of the system. 


UC1 


Number of Use Cases in the system. 


UC2 


Number of communications among UC and Actors 


UC3 


Number of communications among UC and Actores without redundancies 


UC4 


Global complexity of the system 



Table 3. Marchesi Use Case Metrics 



The UC4 metric represents a balance of the global complexity of the system, 
and its value is obtained through the values of UC1, UC2 and UC3 metrics. 



2.4 Other Diagram Metrics 

Statechart diagrams illustrate the behavior of an object. They define different 
states of an object during its lifetime, which are changed by events. A state 
expresses an action of an object during a certain time, when it does not receive 
external stimulus nor is there any change in its attributes. 

Measures like the Number of Entry Actions, Number of Exit Actions, Number 
of Transitions, or even the Number of Activities are associated to the complexity 
and dimension of the problem GMP02 . In Table|4]we can notice some examples 



from measurable attributes for this type of diagram. 

Activity diagrams describe work flows and are very useful for detail opera- 
tions of a class (including behaviors expressed by parallel processing). As we can 
see in Table [5] several metrics for this diagrams are available. 



Statechart Metrics 



Metric 


Description 


TEffects 


Number of transitions with 
an effect in the state ma- 
chine. 


TGuard 


Number of transitions with a 
guard in the state machine. 


TTrigger 


Number of triggers of the 
state machine transitions. 


States 


Number of states in the state 
machine. 



Table 4. Statechart Diagrams Example 



Activity Metrics 



Metric 


Description 


Actions 


Number of activity actions. 


Object- 


Number of activity object 


Nodes 


nodes. 


Pins 


Number of pins on the activ- 
ity nodes. 


Guards 


Number of guards defined on 
object and control flows of 
the activity. 



Table 5. Example of Activity Diagrams 



Besides these metrics, it is possible to measure attributes like the Number 
of Activity Groups/Zones, the Number of object flows or even the Number of 
Exceptions of each diagram. 

After this methodological research through which we introduce the more 
consistent and relevant metrics found in this area, we present in the next section 
tools for apply them to U M L models and put them to test with a real case-study. 



3 Tools for UML Metrics Calculation 

Nowadays, it is very common to use tools like Visual Paradigm for UMU[] or even 
Poseidon for UML^Jfor software application development. They offer a visual 
environment to model software, which reduces the complexity of software design. 
However, they do not support metrics specification - it is necessary to use other 
tools, designed for this task. In the next subsections, we will introduce two 
systems for quantitative analysis of the structural properties of UML models, 
and put them to test for exploring their features with a real case-study. 

One of the tools that we are going to address is SDMetric^J a design mea- 
surement tool for UML models. Although its core is open source and available 
under the GNU Affero General Public License, SDMetrics GUI it is not freely 
distributed. It is a very complete design measurement tool, analyzing a wide 
range of UML diagrams, including Class, Use Case, Activity and Statemachine 
diagrams, generating several metrics for each type of diagram. 

The other tool we test is the Sparx System Enterprise Architect]^] a team-based 
modeling environment. It embraces the full product development lifecycle, sup- 
porting both software design, requirements management, and metrics calculation 
for Use Case Diagrams. It allows to estimate the complexity of the project in an 
earlier stage, as well as the complexity associated with each system actor. 



Available at 
Available at 



http: //www. visual-paradigm. com/product/vpuml/ 



http : / /www . gentleware . com/products . html 



3 Available at http : / /www . sdmetr ics . com/ 
Available at http://www.sparxsystems.com.au 



Case Study 

Our case-study is the model built to describe an information system for the 
MWK (Manages With Knowledge) service management companjFl 



■Com pany Manager 



-hist : String 

-cad Hi story : String 



Client: String 
lUser : String 
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string | 



word : String 
: String 



Fig. 1. Excerpt of a Class diagram 



In order to meet its clients needs MKW is a company that has a wide range 
of suppliers subcontracted to be responsible for services execution. Multiple sup- 
pliers can supply the same service and each service can be delivered in different 
ways. Each service can then be composed of multiple activities. As an example, 
there could be a service called Shirts until 10 Kg and inside this service there 
could be activities such as wash, iron, sewing buttons, etc. Each activity of a 
given service as a stipulated price, and can be hired by a client. 

The complete MKW model is composed of Use Case, Class, Sequence and 
Statemachinc diagrams. As an example of the UML diagrams used to model this 
task, we can see in Figure [2] an image of a general Use Case diagram and, in 
Figure [TJ an excerpt of a Class diagram. 

3.1 SDMetrics 

SDMetrics is a design measurement tool for analyze a wide range of UML dia- 
grams, including Class, Use Case, Activity and Statemachine diagrams. 

5 This modeling project was developed in the context of the Software Systems Devel- 
opment master course 



Manages With Kncwledge 




Fig. 2. The general Use Case model 



For each type of diagram, this tool generates several metrics. For example, the 
NumAttr metric is calculated from Class diagrams and measures the number 
of attributes in a class. Other one is ExtPts, which is calculated from Use Case 
diagrams, and gives us the number of extension points of a given use case. 

SD Metrics is written in Java, and provides us a graphical user interface for 
analyze XM0 files, most modeling tools support project exportation in XML 

This tool allows to access the results from different views. We will introduce 
the ones that seem the most important: 

— Metric Data Tables provides a table that matches each UML model ele- 
ment analyzed (table line) to its value for each metric (tabic column); 

— Histograms provides a graphical distribution for each design metric; 

— Design Comparison provides us a mean to compare the structural prop- 
erties of two UML designs. It is very useful to compare the same design in 
different iterations of the development, or to compare an alternative design 
to the current one. 

— Rule Checker design rules and heuristics detect potential problems in the 
UML design such as incomplete design (i.e. unnamed classes, states with- 
out transitions, etc.); violation of naming conventions for classes, attributes, 
operations, packages; etc; 

— Catalog this view provides us with the definitions of the metrics, design 
rules, and relation matrices for the current data set, and provides literature 
references and a glossary for them. 



XMI (XML Metadata Interchange) is an OMG (Object Management Group) stan- 
dard to generate XML-based representations of UML and other OO data models. 



One of the most advanced features in this software is the possibility of defining 
Custom Design Metrics and Rules. The new metrics are defined in a XMLfilc, 
with a very particular format, the SDMetricsML (SDMetricsMarkup Language). 

The SDMetrics tool does not provide a direct notion of good or bad quality 
of the design model. Despite that, on its User ManuaQwe can find tips of how 
to interpret each kind of metrics. 

Results Based on the SDMetrics manual, we will now explain how to analyze 
each metric obtained. Figure [3] illustrates some outputs of SDMetrics for our 
case-study. On the left, we can see an Histogram for class diagrams evaluating 
the NumAttr metric. On the right, we present an excerpt of a general metrics 
table for class diagrams. 
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Fig. 3. SDMetrics: NumAttr Histogram and Metrics Table for Use Case Diagrams 



Size metrics usually count elements inside design elements (class, package, 
etc). They are good to estimate developing costs and effort, and can be directly 
found on the Metrics Table. A large design element may indicate that it suffers 
from poor design, resulting in low functional cohesion. This has a negative impact 
on the understandability, reusability, and maintainability of the design element. 

Measure Coupling is estimate the degree of elements connection in a design: 
the more they are connected, the more they depend on each others. Changing 
a design element may force the user to adapt the connected elements; also a 
problem in a design element may cause failure in a completely different connected 
element. An high degree of dependency may affect the system maintainability 
and testability - it is crucial to minimize coupling. 

Inheritance-related metrics usually calculate features such as depth/width of 
the inheritance and number of ancestors/descendants of a design element. Such 
as high coupled elements, deep inheritance structures are believed to be more 
fault-prone. It is harder to fully understand a class that is situated deep in the 
inheritance tree, because you have to understand its ancestors. Also, modifying a 
design element with many descendants, may have a large impact on the system. 

Complexity metrics measure the degree of connectivity between elements of 
a design element. They are concerned with relationships/dependencies between 



7 



http : //www . sdmetrics . com/mamial/index .html 



the elements in the design unit, such as the number of method invocations inside 
a class. The high complexity between the elements of a design element can make 
the design harder to understand, therefore more propitious to faults. Complexity 
metrics are usually strongly correlated with size measures. So even though they 
are good indicators of quality, such as fault-proneness, they provide no new 
knowledge comparing to size metrics. 

In general, these guidelines lead us to belief that the lower the metrics values 
are, the better. In what concerns our case-study, after observed the metrics 
obtained we have noticed that GCS class seems to have too many operations 
(26), specially compared with the rest of the classes. On the other hand, some 
classes look like they are missing operations. These conclusions suggest a careful 
analysis of the project to identify classes which operations do not belong to them 
and should be given to other classes; also classes that lack operations, should 
be completed. This is just an example of how the output of SDMetrics could be 
useful during a modeling process. 

3.2 Sparx Systems Enterprise Architect 

Sparx Systems Enterprise Architect is another tool that provides modeling of 
UML diagrams. It supports mind map diagrams and project management, to 
provide full traceability from requirements specification to deployment end im- 
plementation. This tool also provides some metrics evaluation to compute the 
complexity of a project based on Use Case diagrams. 

To perform this evaluation, the user needs to provide a level of implementa- 
tion complexity for each interaction with authors. This task can be done when 
defining the Use Case descriptions or when performing the metrics evaluation. 

To evaluate the metrics, Enterprise Architect has a wizard which enables other 
options for the complexity analysis. These options manipulate the Technical and 
the Environment complexities and are used to adapt the evaluation and perform 
a better result on estimation. 

This system enables filtering the Use Cases used in evaluation, both on man- 
ual selection or regular expressions over use case name. This kind of filtering 
enables the project to be distributed and evaluated individually. 

Results For use this tool in metrics calculation, the user have the possibility 
to do some tweaking over use case complexity (on their description) to get more 
precise results. This complexity admits simple values like low, medium or high. 

Enterprise Architect offers a wizard which enables the edition of factors related 
to environment and technics complexity or even the hour rates, as we can see 
on the left side of Figure |4j Here we can use multiple factors to evaluate this 
factors, like usability or portability on technical complexity. 

We can also access a metrics wizard, as depicted in the right side of Figure 
[2J which illustrates the set of default values used to evaluate the task effort. A 
set of predefined values are based on the factors edited on the previous wizard. 
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Fig. 4. Estimation Factors and Use Case Metrics of Enterprise Architect wizards 



The final result of the metrics evaluation process is an estimation of Working 
Hours, Use Case Points RibOl and Total Cost needed to perform the system 
development. In our case-study, the project has twelve Use Cases and many of 
them have medium complexity. Thus, as we can see in the right side of Figure [4j 
the effort to complete the task is 552 working hours, that would give a final result 
of € 5.520. For obtaining this value only was changed the use cases complexity 
and everything else was left with default values. 



4 Metrics Assessment 



When interpreting the values obtained from measurements, one might question: 
is the metric really measuring the intended attribute? This is a question that is 
present in the industry, yet unsuccessfully answered. 

Working with models, one might want to know the quality of its model, i.e., 
which amount of it really reflects object proprieties. To discuss model quality, 



one must use metrics to quantify those proprieties. Fenton Fen99 estimates that 



companies spend about 4% of the development budget in the establishment of 
metrics programs, therefore, engineers should also guarantee that the applied 
metrics actually quantify, measure, and model the attributes of the system. 



Kaner and Bond |KB04 proposed a framework for metric evaluation, saying 



that if a project or company is managed according to the results of inadequately 
validated metrics, and not tightly linked to the attributes they are intended to 
measure, distortions and disfunctionalities should be commonplace. 

The industry, although not having a formal answer to this question, has ad- 
vanced some steps forward in this direction. The IEEE Standard 1061 IEE98 



defines an attribute as "a measurable physical or abstract property of an en- 
tity". A quality factor is a type of attribute, "a management- oriented attribute 
of software that contributes to its quality". A metric is defined as being a mea- 
surement function, and a software quality metric is defined as "a function 



whose inputs are software data and whose output is a single numerical value that 
can be interpreted as the degree to which software possesses a given attribute that 
affects its quality". Any software metric must comply with the following criteria: 
correlation, consistency, tracking, predictability, discriminative power and relia- 
bility. This provides a sound layout of a methodology for developing metrics for 
software quality attributes. 

5 Conclusion 

In general, the software development process follows a systematic approach aim- 
ing at a product/system of quality. The quality criterion is not limited to the 
attributes of the final product, whether they comply with industry standards or 
not; it also ensures that the software fulfills all the specified requirements. Most 
of the existing approaches that include metrics on the software lifecycle involve 
source code analysis and cannot be applied in earlier stages of the development 
process. Applying metrics to UML models enables the estimation of development 
effort at an earlier stage, as well as implementation time, complexity and cost 
of the system under development. 

In this paper we focus on the most relevant metrics for UML models and on 
two tools capable of measuring them: SDMetrics and Enterprise Architect systems. 
We believe that our research gathers the more consistent and relevant metrics 
for assessing the quality of UML models. The paper aims at offering an overview 
of the advantages of an early estimation of the process efforts, and to provide 
guidelines to the most important works on this area Combined with a study of 
UML metrics extraction tools, it represents an important support for an end user 
which needs to pick up a tool that best suits its needs. 

We can conclude that SDMetrics is a versatile tool to calculate a large set of 
metrics over a wide range of UML diagrams. Based on this metrics we can try 
to measure the quality and complexity of a software model. It has an interesting 
GUI which provides several output views, from simple tables to Histograms. It 
finds potential problems with the model and also enables new metrics definition. 

The major disadvantage of this tool is the incapability of giving the user 
a plain and simple notion of the model quality, although SDMetrics Manual 
provides simple tips of how to interpret each kind of metric - crucial for a correct 
results reading. At a glance, SDMetrics results are guidelines for finding the good 
and the bad points of U M L models, not a full specification of the models quality. 

On the other hand, Enterprise Architect is a full formal UML specification 
environment that supports metrics calculation oriented to Use Case diagrams. 
It is driven to enterprise market and oriented to minimize the cost and time spent 
with the production process. This tool provides simple results and represents a 
good choice to have an estimation of the implementation costs based on a formal 
system specification. With its system wizard, the user can adapt the value of 
the factors related to the environment and technics complexity, to obtain an 
accurate estimation of the system final cost. This requires extra user interaction 
in the specification of a task complexity and a large knowledge of the system 



under development. This complexity could be archived by other means if other 
types of diagrams were also analyzed. Summing up, Enterprise Architect does 
not provide any kind of quality model analysis or do any automatic system 
complexity evaluation: it estimates the final cost and effort of the project. 
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